Image sensing control method and apparatus, image transmission control method, apparatus, and system, and storage means storing program that implements the method

ABSTRACT

This invention relates to an image sensing control method and apparatus, and an image transmission control method, apparatus, and system, in which a camera such as a video camera is connected, the operation of the camera is controlled in accordance with a control command sent from a client via a network, and an image signal sensed by the camera is transmitted to the client via the network. Upon reception of a control command of the pan and tilt angles, zoom value, or the like of the camera from the client, the position control of the camera is done in accordance with the command, and transmission of the image signal sensed by the camera is suspended until the instructed operation is complete. Alternatively, upon reception of such control command, the image signal sensed by the camera is compressed at a high compression ratio, and the compressed image signal is transmitted. On the client device, received image signal is directly displayed without buffering during the control operation, thus allowing real-time image display during the position control of the camera.

BACKGROUND OF THE INVENTION

The present invention relates to an image sensing control method andapparatus for remote-controlling an image sensing device such as a stillcamera, video camera, and the like, an image transmission controlmethod, apparatus, and system for transmitting the sensed image, and acomputer readable storage medium that stores a program for implementingthe method.

A camera control system in which a plurality of cameras each having panand tilt functions and a zoom function are set at the respectivepositions, and an operation device at the central position or anarbitrary position is connected to these cameras via a dedicatedcommunication line to remote-control an arbitrary camera, is known. Insuch system, control signals (cables or signal lines) for transmittingcamera control/status signals are required in addition to cables fortransmitting video/audio signals. Also, a method of modulating thesecamera control/status signals and superposing them on video/audiosignals is known. Theoretically speaking, however, such method isequivalent to a method of assigning separate communication channels toindividual signals.

In such camera control system (a system that remote-controls camerasusing the dedicated communication line), an apparatus forremote-controlling a camera is fixed in position, and becomes abottleneck against easy system expansion.

For this reason, it is preferable to remote-control cameras on a networkbuilt by computers and the like. This is because both an apparatus forcontrolling a camera and an apparatus for remote-controlling a cameraoperate as members of a computer network (to be simply referred to as anetwork hereinafter), and users can freely remote-control theseapparatuses from terminals (e.g., personal computers or the like)connected to the network.

On the other hand, a camera conference system that transmits audiosignals onto the network in addition to images sensed by the cameras isreceiving a lot of attention.

FIG. 1 schematically shows a general image communication systemincluding the above-mentioned camera conference system. FIG. 2 is aschematic block diagram showing the arrangement of the imagecommunication system shown in FIG. 1. A communication terminal 100serves as an image server for controlling, e.g., a video camera andtransmitting the image sensed by the camera comprises a computer mainbody 101, a keyboard 102, a mouse 103, a display 104, and a video camera105, as shown in FIG. 1. A communication terminal 200 serving as animage client which receives image data from the communication terminal100 and displays the received image comprises a computer main body 201,a keyboard 202, a mouse 203, and a display 204, as shown in FIG. 1. Thecommunication terminals 100 and 200 (client) are connected via a network300 to be able to communicate with each other.

As shown in FIG. 2, the image sensed by the video camera 105 of thecommunication terminal 100 is converted into digital data by an imageinput unit 110. The digital image data for one frame is packetized by anetwork transmitter/receiver 111, and the obtained packet is output ontothe network 300. The packet output onto the network 300 is received by anetwork transmitter/receiver 211 of the client (communication terminal200). Upon reception of the packet, the client converts the receivedpacket into data, and stores the data in an image buffer 210. The clientreads out the image data from the image buffer 210 at a predeterminedtiming, and displays the image on the display 204.

By repeating the above operation, moving images captured by thecommunication terminal 105 via the video camera 105 can be displayed onthe client side. The image data is stored in the image buffer 210 forthe following reason.

A packet on the network has fluctuation (jitter) with respect to timefor various reasons. FIG. 3 shows this state. As shown in FIG. 3, evenwhen the transmitter side of image data transmits the image data for oneframe at a predetermined interval T, the receiver side does not alwaysreceive the image data at the predetermined interval T due to theinfluence of the jitter. For this reason, the reception interval maybroaden, as indicated by A in FIG. 3, or may narrow, as indicated by Bin FIG. 3.

Hence, if the client displays the received data immediately withoutgoing through the image buffer 210, the motion of images displayed onthe display 204 may be too slow (the portion A) or too fast (the portionB). In order to prevent such irregular motion, image signals for a timethat can absorb the jitter are normally buffered in the image buffer210, and the stored signals are read out at the same timing as that uponcapture of the images in the image reproduction mode, thus removing theinfluence of the jitter.

However, this conventional system does not consider any case wherein thevideo camera 105, the pan, tilt, zoom functions and the like of whichcan be externally controlled, is connected to the terminal 100, and theposition of the video camera 105 is controlled by a client on which animage is being displayed. This problem will be explained in detailbelow. In the conventional system, since the image data to be displayedare buffered by the receiving communication terminal 200, the imageacquisition time has a large difference from the image display time. Forthis reason, when the client on which an image is being displayedcontrols the pan, tilt, or zoom function of the video camera 105 whileobserving the displayed image, a certain lag time is produced from whenthe user controls a function of the video camera 105 until the imagecorresponding to the controlled function is displayed on the clientside. Hence, the user controls the video camera 105 more than he or shewanted, and e.g., the field of view of the video camera 105 moves beyondthe angle that the user intended.

A problem posed when the angle (including pan and tilt angles and zoomratio) of the camera has changed will be explained below.

For example, as shown in FIG. 4, assume that a camera which points in adirection of an object A is panned to sense the image of an object B. Inthis case, since the transmission data volume is constant, images arecaptured at a predetermined frame rate and image data is kepttransmitted to the client while the camera angle is being changed fromthe object A to the object B.

Such images transmitted during the change from the object A to theobject B may be desirable in some respects, but are not so significantfor the user who knows the positional relationship between the objects Aand B. In fact, such user may feel that nonsense images aretransmitted/received.

When the above-mentioned camera control communication terminal (cameracontrol device) and client are built on the network, transmission ofunwanted information is preferably precluded, since it influences theoperability of other users on the network.

In particular, the “Internet” is receiving a lot of attention. Whencamera control devices and clients are built on such huge network, theabove-mentioned problem comes into focus inevitably.

In the system that transmits images using digital transmission mediasuch as the Internet, in general, the sensed digital image data iscompressed by a compression technique, e.g., Motion JPEG, MPEG, or thelike, and the compressed image data is transmitted. The compressed imagedata is expanded by a device (to be referred to as a viewerhereinafter), and thereafter, the expanded image is displayed on thedisplay.

However, in such conventional system, it is a common practice not toperform compression processing upon transmission of images, or topermanently use a single compression scheme suitable for image contentsor network characteristics (transmission rate or the like), ifcompression processing is done. For this reason, when an uncompressedimage signal is transmitted from a network with a low transmission rateor when the frame rate drops since an image is compressed by acompression scheme suitable for a high-resolution image (a scheme with alow compression rate), if the user makes camera control such as panningor tilting, images are displayed intermittently. As a result, the userfails to recognize the current state (e.g., pan or tilt angle) of thecamera which is sensing an image in real time.

In such image transmission system, the images to be transmitted to allthe clients are subjected to identical compression or image processingindependently of whether or not the client who receives media data hasthe right of access to that video camera. That is, an identical mediaprocessing method is used for all the clients. For this reason, evenwhen the client who has the right of camera access wants to observe animage with higher resolution so as to control the operation of thecamera, he or she can only observe the image with a normal resolution.On the other hand, a client who has no right of camera access need notobserve an image with a high resolution, and wants to display images inreal time at high speed. However, such switching cannot be done in theconventional system.

On the client, image packets are stored using the image buffer (jitterbuffer), and frame images to be displayed are generated using the storedpackets, as described above. On the other hand, in order tosatisfactorily attain camera control at the client having the right ofcamera access, images are preferably reproduced in real time so as torecognize images of the sensed object in real time.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theabove-mentioned problems, and has as its object to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which suppress the transmission volume ofsensed image data until an image sensing angle, magnification of animage sensing device reaches a desired angle, so that a desired imagesensing angle by an operator can be quickly obtained, and the load onthe network during changes in angle and/or magnification can be reduced.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which change a compression method of an imagesignal during operation control of an image sensing apparatus, andtransmit the image signal compressed by the changed compression methodonto the network, so that changes in image by the operation control ofthe image sensing device can be detected in real time via the network.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which increase the compression ratio of animage signal sensed by an image sensing device when the operation of theimage sensing device is controlled, and transmit the image signalcompressed at a high compression rate, so that the time required fortransmitting the image signal can be shortened, and any lag of imagesignals sensed by the image sensing device during the operation controlcan be reduced.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which can reduce any lag between the time atwhich the transmitter acquired images by controlling an image sensingdevice and the time at which a client receives and displays that imagesignal upon remote-control of the image sensing device, and can improvethe response of displayed images with respect to the operation state ofthe image sensing device when the client remote-controls the imagesensing device while observing the displayed images.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which can quickly display image signalsacquired by a control device after an operation of an image sensingdevice when a client remote-controls the image sensing device via acontrol device of the image sensing device.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which change the processing method of an imagesignal to be transmitted to a client depending on whether or not theclient who receives an image signal sensed by an image sensing devicevia a network has the right of access to the image sensing device.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, which transmit an image signal which isconverted into low-compression codes, to a client who has the right ofaccess to an image sensing device, and transmit an image signal, whichis converted into high-compression codes, to a client who has no rightof access.

It is another object of the present invention to provide an imagesensing control method and apparatus, image transmission control method,apparatus, and system, and a storage medium that stores a program forimplementing the method, in which a client who has the right of accessto an image sensing device, comprises a small-capacity jitter absorptionbuffer, and a client who has no right of access comprises alarge-capacity jitter absorption buffer.

Other features and advantages of the present invention will be apparentfrom the following descriptions taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the descriptions, serve to explain the principle of theinvention.

FIG. 1 is a schematic diagram showing a general image communicationsystem;

FIG. 2 is a schematic block diagram showing the arrangement of thegeneral image communication system;

FIG. 3 is a chart for explaining jitter produced in communications onthe network;

FIG. 4 is a view showing the state of changing objects of a camera;

FIG. 5 is a block diagram showing the arrangement of an imagecommunication system according to the first embodiment of the presentinvention;

FIG. 6 is a view showing an example of a camera console window of thefirst embodiment;

FIG. 7 is a view showing an example of an image display screen of thefirst embodiment;

FIG. 8 is a view showing a camera table and a connection/disconnectionmenu in the first embodiment;

FIG. 9 is a flow chart showing the operation processing contents of acamera operation of a camera operation device of the first embodiment;

FIG. 10 is a flow chart showing the reception processing contents ofimage data in the camera operation device of the first embodiment;

FIG. 11 is a flow chart showing the operation processing contents of acamera management device in the first embodiment;

FIG. 12 is a view showing display examples of the individual angles insimple angle movement in the first embodiment;

FIG. 13 is a block diagram showing a modification of the system of thefirst embodiment;

FIG. 14 is a block diagram showing the arrangement of an imagetransmission system according to the second embodiment of the presentinvention;

FIG. 15 is a flow chart showing the flow of image data in the imagetransmission system of the second embodiment;

FIG. 16 is a chart showing the flow of control commands in the imagetransmission system of the second embodiment;

FIG. 17 is a flow chart showing the processing of a camera controller ofthe image transmission system of the second embodiment;

FIG. 18 is a block diagram for explaining the arrangement of an imagetransmission system according to the third embodiment of the presentinvention;

FIG. 19 is a view showing a display example at a communication terminalof the third embodiment;

FIG. 20 is a block diagram showing the arrangement of an image bufferand its peripheral portions in the third embodiment in detail;

FIG. 21 is a flow chart showing the display sequence of received imagedata at a communication terminal of the third embodiment;

FIG. 22 is a block diagram showing the arrangement of an imagecommunication system according to the fourth embodiment of the presentinvention;

FIG. 23 is a block diagram showing the arrangement of an image bufferand its peripheral portions in the fourth embodiment in detail;

FIG. 24 is a flow chart for explaining the processing sequence of abuffer clearing unit in the fourth embodiment;

FIG. 25 shows a memory map example of a storage medium that stores acontrol program according to the third and fourth embodiments of thepresent invention;

FIG. 26 is a block diagram showing the arrangement of an imagecommunication system according to the fifth embodiment of the presentinvention;

FIG. 27 is a block diagram showing the arrangement of a video captureunit of the fifth embodiment;

FIG. 28 is a block diagram showing the arrangement of a client deviceaccording to the sixth embodiment of the present invention;

FIG. 29 shows the format of a transmission destination table of thefifth embodiment;

FIG. 30 shows the data format of image data of the fifth embodiment;

FIG. 31 is a flow chart showing the processing at a camera server deviceof the fifth embodiment; and

FIG. 32 is a flow chart showing the processing at a client deviceaccording to the sixth embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The preferred embodiments of the present invention will be described indetail hereinafter with reference to the accompanying drawings.

[First Embodiment]

FIG. 5 is a schematic block diagram showing the arrangement of an imagecommunication system according to the first embodiment of the presentinvention. In this embodiment, camera management devices and cameraoperation devices (clients) are connected via a LAN but may be connectedvia a communication network such as the Internet.

Reference numeral 10 denotes a LAN adopting Ethernet (a LAN with a busconfiguration developed by DEC Corp., Intel Corp., and Xerox Corp.,USA), to which a plurality of camera management devices 12 and 14 areconnected. The camera management devices 12 and 14 are respectivelyconnected to video cameras (to be referred to as cameras hereinafter) 16and 18. The cameras 16 and 18 may have microphones for inputting audiodata. Furthermore, a plurality of camera operation devices 20 and 22 forremote-controlling the cameras 16 and 18 are connected to the LAN 10.

As will be described in detail later, the camera management device 12 or14 controls the image sensing direction (at least one of pan and tiltdirections) and zoom ratio of the camera 16 or 18 respectively, underits management in accordance with a camera control signal supplied fromthe camera operation device 20 or 22 via the LAN 10, and transmit aimage signal obtained from the camera 16 or 18 to clients as the supplysources of the control signal via the LAN 10. Note that when thesecameras 16 and 18 comprise microphones, audio information is alsoconverted into predetermined data, which is transmitted to thecorresponding client.

The camera management device 12 comprises a communication controller 30,a storage unit 32, a camera controller 34, and an interface 36. Thecommunication controller 30 controls connections and data communicationswith an arbitrary camera operation device 20 or 22 via the LAN 10. Thestorage unit 32 comprises a memory such as a RAM, an HDD (hard diskdrive), or the like, and stores various data including image and audiodata to be transmitted. Also, the storage unit 32 stores positioninformation of an object to sense a specific object. Note that thisposition information includes the image sensing direction and zoom ratioof the camera 16. The camera controller 34 controls the image sensingdirection and zoom ratio of the camera 16 and supplies image and audiodata from the camera 16 to the communication controller 30 via thestorage unit 32. The interface (I/F) 36 is used for connecting thecamera 16 to the camera controller 34. Note that the camera managementdevice 14 also has the same arrangement as that of the camera managementdevice 12.

On the other hand, the camera operation device 20 comprises acommunication controller 40, a storage unit 42, a camera selector 44, aconsole 46, an operation manager 48, a display monitor (to be referredto as a monitor hereinafter) 50, and a loudspeaker 52. The communicationcontroller 40 controls connections and data communications with anarbitrary camera management device 12 or 14 via the LAN 10. The storageunit 42 comprises a memory such as a RAM, an HDD (hard disk drive), orthe like, and stores various data including image and audio datareceived via the LAN 10. The console 46 operates the pan and tilt anglesand zoom ratio of one camera selected by the camera selector 44. Theoperation manager 48 manages selection by the camera selector 44 anduser's operations at the console 46, and exchanges control informationwith the camera management device 12 or 14 that manages the camera to becontrolled. The monitor 50 displays an image received from the cameramanagement device via the LAN 10. The loudspeaker 52 acousticallyoutputs a voice received via the LAN 10.

Note that the camera operation device 22 has the same arrangement asthat of the camera operation device 20. The camera operation devices 20and 22 can substantially use personal computers, workstations, or thelike, and the camera selector 44, console 46, and operation manager 48are implemented by image elements or selection menus, and the like seton a graphical user interface (GUI), a pointing device such as a mousefor operating or selecting them, and a software program that displaysthe GUI and processes the operation or selection of the pointing device.

FIG. 5 illustrates two camera management devices 12 and 14 and twocamera operation devices 20 and 22. As is apparent, three or more cameramanagement devices can be connected to the LAN 10, and similarly, threeor more camera operation devices to the LAN 10. In the followingdescription, a case will be exemplified wherein the camera operationdevice 20 controls the camera 16 via the camera management device 12. Ofcourse, the present invention can be applied to a case wherein thecamera operation device 22 can control the camera via the cameramanagement device 12 or 14.

FIG. 6 shows a camera console window 60 depicted as the console 46 inFIG. 5, and this camera console window 60 is displayed on the screen ofthe monitor 50. The camera console window 60 has pan buttons 62 and 64for instructing the pan direction (horizontal direction) of the camera16, tilt buttons 66 and 68 for instructing the tilt direction (verticaldirection) of the camera 16, a home button 70 for returning the camera16 to a predetermined front position, a telephoto button 72 forinstructing the camera 16 to zoom in the telephoto direction, and awide-angle button 74 for instructing the camera 16 to zoom in thewide-angle direction. Furthermore, the window 60 has buttons 61, 63, 65,67, 69, and 71 for instructing the image sensing directions and zoomratios of specific objects to be sensed by the camera 16. The buttons61, 63, 65, 67, 69, and 71 respectively have independent image sensingdirections and zoom ratios, and data associated with them are stored inthe storage unit 32 of the camera management device 12.

A rectangular frame 78 indicates a view field range at the current panangle, tilt angle, and zoom value of the camera 16, and a range 76indicates the range (to be referred to as a potential view field rangehereinafter) that allows image sensing when the pan and tilt angles aremaximized while maintaining that zoom value.

FIG. 7 shows an image display window 77, which is displayed on thescreen of the monitor 50 in addition to the camera console window 60(FIG. 6). Within an image display frame 79 assured on the image displaywindow 77, an image sensed by the camera 16 is displayed. The operatoroperates the camera 16 while checking the motion of the object byobserving the image displayed on the image display frame 79. Note thatthe image display frame 79 may be assured within the rectangular frame78 (FIG. 6).

On the image display window 77, “entrance camera 1” displayed on theupper portion indicates a camera which is currently sensing an imagethat is being displayed on the image display frame 79, i.e., thecurrently selected camera. Furthermore, “15 fps” displayed on the lowerportion of the image display window 77 indicates the current frame rate,and is 15 frames/sec in the illustrated example. Also, “290 kbps”indicates the current data transmission rate, and is 290 kbits/sec inthis case. Since these two pieces of information are displayed together,the user's operability can be improved.

FIG. 8 shows a camera selection window 80 for selecting one of thecameras to be operated which are included in the camera selector 44 inFIG. 5, and are displayed. This window 80 is also displayed on thescreen of the monitor 50. A “connect” pull-down menu 81 has items“connect” and “disconnect”. Together with the display of the pull-downmenu 81 upon selection of the “connect” menu or in accordance with theselection of a camera list display menu (not shown), a camera table 82that shows a list of all the selectable cameras is displayed.

The camera table 82 holds a combination of names or the like that easilyrepresent the individual camera positions (“entrance camera 1” is one ofthem), and identification information for uniquely identifying thecamera management device 12 or 14. In the first embodiment, since thecamera management devices 12 and 14 and the camera operation devices 20and 22 are connected via an IP (Internet Protocol), an IP address isused as this identification information. The camera names and the IPaddresses of the camera management devices (corresponding to the cameratable 82 in FIG. 8) are pre-stored in the storage unit 42.

When a camera is selected from those displayed on the camera table 82,and item “connect” is selected from the “connect” pull-down menu 81, theselected camera can be controlled via the LAN 10 by the camera operationdevice 20, and an image sensed by the selected camera is displayed onthe display frame 79 of the monitor 50.

When one already connected camera is selected from those displayed onthe camera table 82, and item “disconnect” is selected from the“connect” pull-down menu 81, the right of access to that camera isreleased to another camera operation device, and an image sensed by thatcamera ceases to be displayed on the monitor 50.

Note that a plurality of camera operation devices may try to control acamera connected to one camera management device. In such case, theright of access is granted to only the camera operation device which wasconnected earliest. Note that the right of access is not granted toother camera operation devices, but the image sensed by that camera canbe observed at these devices. Since other camera operation devices haveno right of access to that camera, various operation buttons shown inFIG. 6 are displayed at, e.g., a low density, to indicate an inactivestate, and even if the user tries to operate them, the operations areignored. If the user of the camera operation device with the right ofaccess disconnects that camera, the right of access is granted toanother camera operation device which has not been granted the right ofaccess so far, and the above-mentioned buttons are displayed on themonitor of the camera operation device which is newly granted the rightof access in an active state. The order of granting the right of accessof the camera corresponds to the order of connections to the cameramanagement device connected to the camera.

A case will be exemplified below wherein the camera operation device 20receives and displays an image from the camera 16 connected to thecamera management device 12.

FIG. 9 is a flow chart showing the processing at the camera operationdevice 20 of the first embodiment.

As described above, since the hardware portion of the camera operationdevice 20 comprises a normally used personal computer or workstation,the console 46, the camera selector 44, and the operation manager 48 areimplemented by a CPU of the device, and their operation program isloaded from an external storage device (not shown) onto a main memory.The following description will explain the processing after the cameraoperation program has already been read out and executed.

The operator clicks one of the operation buttons 61 to 74 on the cameraconsole window 60 (FIG. 6) using a pointing device (not shown). Theoperation manager 48 (FIG. 5) detects the already connected camera (stepS11), and if none of cameras are connected, the flow advances to stepS12 to display a message (non-connection message) on the monitor 50,thus ending this processing.

On the other hand, if the camera operation device is already connectedto one of the camera management devices (in this case, the cameramanagement device 12), the flow advances to step S13 to check if theright of access has been granted by the connected camera managementdevice 12. If NO in step S13, the flow advances to step S14 to display amessage (control denial message), thus ending this processing.

If it is determined that the camera operation device is alreadyconnected to the camera management device 12 and has been granted theright of access, the flow advances to step S15 to acquire the clickedbutton status.

In the following description, the operation at the button 62, 64, 66,68, 70, 72, or 74 in FIG. 6 will be referred to as normal control, andthe operation at the button 62, 63, 65, 67, 69, or 71 will be referredto as simple control.

It is checked in step S16 if clicking instructs the normal or simplecontrol.

If the normal control is instructed, the flow advances to step S17. Inthis case, for example, when the instruction button 64 for increasingthe pan angle is operated, a control command for increasing the panangle is generated.

On the other hand, if the simple control (input at the button 61, 63,65, 67, 69, or 71) is instructed, the flow advances to step S18, and acontrol command having identification information of the instructedbutton is generated.

After the control command is generated, the flow advances to step S19 totransmit the generated control command to the connected cameramanagement device 12.

It is checked in step S20 if the camera is in operation. If the camera16 is in operation (this means that one of various buttons is keptpressed), the flow returns to step S15 to repeat the above-mentionedprocessing.

FIG. 10 shows the sequence of the program executed by the CPU of thecamera operation device 20 upon reception of image data from the cameramanagement device 12.

Upon reception of image data, header information of the received data isacquired in step S31, and the following image data is received andexpanded in step S32. The flow then advances to step S33, and theposition and size of the frame 78 in FIG. 6 are updated on the basis ofangle information (pan and tilt angles and zoom value) included in theacquired header information Thereafter, the expanded image data isdisplayed within the image display frame 79 shown in FIG. 7 in step S34.

Since the camera console window 60 is updated in step S33, the cameraconsole window 60 to be displayed on the camera operation device with noright of access can be matched with that displayed on the cameraoperation device with the right of access.

The operation processing sequence in the camera management device 12 ofthis embodiment will be described below with reference to the flow chartin FIG. 11. Note that the program corresponding to this flow chart, ofcourse, is stored in a predetermined storage medium of the cameramanagement device 12, and is executed by the CPU (not shown). On theother hand, login processing from the camera operation device, grantingof the right of access, and the like are attained by another processing,and only the operation control of the panning and tilt angles, zoomvalue, and the like of the camera will be described hereinafter.

In step S41, it is checked if the received command is a normal controlcommand. If YES in step S41, the flow advances to step S42 to controlthe pan or tilt angle or zoom value in accordance with the instructedangle information, thus ending this processing. Note that processing foradding the above-mentioned header information to image data input by thecamera 16 in another task and transmitting the data to the connectedcamera operation device 20 is executed during this interval.

If it is determined in step S41 that the received command is not anormal control command, the flow advances to step S43 to check if thereceived command is a simple control command. If NO in step S43, theflow advances to step S44 to execute the corresponding processing.

On the other hand, if YES in step S43, the flow advances to step S45 totemporarily interrupt transmission processing of image data input by thecamera 16 to the camera operation device 20 (also interrupt transmissionof audio data obtained by the microphone if the microphone is connectedto the camera 16). In step S46, the corresponding pan and tilt anglesand zoom value are read out from the storage unit 32 on the basis of theidentification information of the simple control instructed by thereceived simple control command, and the camera 16 is controlled to havethe camera angle defined by these parameters.

In the method of controlling the camera 16, the camera 16 is instructedvia the camera I/F 36 to have the target pan and tilt angles and zoomratio, and the current position information (the current pan and tiltangles and zoom ratio) of the camera 16 is read from the camera 16 viathe I/F 36 and compared with the target values. If the differencebetween the current position information and the target positioninformation becomes smaller than a given value, it is determined thatthe target angle has been reached. Also, the camera 16 may return areturn value indicating an end of the position control.

If it is determined in step S47 that the target angle has been reached,the flow advances to step S48 to restart transmission of video data(including audio data), thus ending this processing.

In the above-mentioned processing, if the normal control command isreceived, even when the camera operation device 20 instructs tocontinuously change the pan angle, the camera operation device 20executes processing for transmitting normal commands to the cameramanagement device 12 at appropriate timings, and receiving image databased on the transmitted commands. On the other hand, the cameramanagement device 12 controls the position of the camera 16 based on theinstructed direction or zoom value every time it receives a command.

On the contrary, when the camera management device 12 receives thesimple control command, image data ceases to be transmitted (imagesensing itself may be disabled) until the target angle has been reached.Hence, the camera can reach the target angle at high speed.

As described above, according to the first embodiment, when the cameraoperation device inputs an angle change instruction using the pan ortilt angle or the zoom value, it can execute a mode of displaying animage in real time in accordance with the instruction, or a mode ofnonstop-moving the camera angle to the target position at high speed andinterrupting image transmission during that movement, thus improving theoperability upon connecting the accustomed camera management device.Also, since substantially no image data is transmitted from the cameramanagement device onto the network until the target angle is reached inthe camera, the load on the accesses of other network users can bereduced.

In the first embodiment, upon reception of the simple control command,image data transmission is interrupted until the target angle is reachedin the camera. Alternatively, the data volume of the image data may bereduced (by increasing the compression ratio). In this case, the datavolume can be reduced to some extent until the target angle is reached,but cannot become substantially zero unlike the above embodiment.

In this embodiment, the camera server, i.e., the camera managementdevice has the pan and tilt angles and zoom values upon transmission ofthe simple control command. However, these values may be freely set atthe client side, i.e., the camera operation device side, and may beheld. If the client has such information, when, for example, a simplecontrol pull-down menu or the like may be displayed, and one of items isselected from the displayed menu, a command with an appropriate format(of course, a format that can be distinguished from the normal controlcommand) may be generated and transmitted to the camera server. In thiscase, the camera server may check the format of the received command todetermine if the received command is the normal or simple controlcommand, and may execute the above-mentioned processing incorrespondence with the received command.

When the client side has angle information used for the simple controlcommand, the angles to be controlled can be standardized in other cameraservers, and need not be stored in units of camera servers.

Independently of the storage position (the camera server or client) ofthe angle information for the simple control, the operator at the clientside preferably can confirm the camera angle set.

For example, the operator can confirm the angles to be set by thebuttons 61, 63, . . . in FIG. 6 when he or she actually operates thesebuttons, but does not know the angles corresponding to these buttonswhen he or she has established a connection for the first time. In somecases, the operator may forget the angles actually set.

In view of this problem, when the camera server has angle informationassociated with the simple control, a button for instructing to transmitthe registered angle information may be arranged, and upon operation ofthis button, a command indicating it is transmitted. Upon reception ofthis command, the camera server transmits the registered angleinformation in units of buttons to the client side in step S44 in FIG.11. Alternatively, the camera server transmits images drawn with framesbased on the angle information.

As a result, the client side displays windows shown in FIG. 12, so thatthe operator at the client can recognize the angles corresponding to thebuttons. Note that reference numeral 90 in FIG. 12 denotes a potentialview field range when panning and tilting are repeated while fixing thezoom ratio of the camera at the wide-angle end.

On the other hand, when the client has the angle information, theregistered information may be read out from the storage unit uponreception of a display instruction of the angle information, and thewindows shown in FIG. 12 may be displayed based on the readoutinformation.

In either case, when the windows shown in FIG. 12 are displayed, theindividual angles can be preferably edited using an appropriate editor.When the camera server stores data for the simple control, the clienttransmits the edit result to the camera server by generating a commandwith an appropriate format. Upon reception of the command, the cameraserver may update its internal data.

On the other hand, when the client stores data for the simple control,it can directly update the stored data.

Note that the first embodiment has exemplified a relatively small-scalenetwork, but may be applied to the Internet that has been receiving alot of attention recently. In this case, remote cameras set at, e.g.,tourist spots (of course, those at foreign countries) can be controlled,and the operator can observe an image of a desired spot at his or herhome. When this embodiment is applied to the Internet, since many andunspecified clients are connected, the simple control camera angle oneach camera server is preferably inhibited from being changed or apermission for changing the angle may be granted to only a specificprivileged user.

When this embodiment is implemented on the Internet, and when a givenclient is connected to a target camera server and issues a properregistration instruction (assigned to, e.g., a pull-down menu) to beeasily connected to that camera server again, the identificationinformation of the camera server connected at that time may beregistered in the camera table (see FIG. 8).

When this embodiment is applied to the Internet, since the client sideruns a browser program, if the window shown in FIG. 6 corresponds tothis browser window, the simple control buttons 61, 63, . . . , aredefined by hypertexts sent from the camera server side. That is, thedisplay of the window shown in FIG. 6 indicates that the camera serverside has information associated with the simple control command, and thebrowser need only inform the camera server of the operated button. Ifthe client side has data for the simple control, the browser adopts,e.g., a pull-down menu.

The first embodiment has exemplified a case wherein the LAN is builtusing the Ethernet, but can also be applied to the Internet, asdescribed above. In this case, the camera servers and clients may beconnected via an ISDN or public telephone network. When the client isconnected to a specific camera server via the ISDN or public telephonenetwork, telephone numbers can be used as information for uniquelyidentifying the camera management devices (camera servers) 12 and 14.

Furthermore, the camera operation device in the first embodiment isrealized by a normally used personal computer or workstation. Also, thecamera management device can be made up of a similar arrangement,needless to say.

FIG. 13 is a schematic block diagram showing the modification of thefirst embodiment. A camera management device 112 is connected to the LAN10, and manages two cameras 114 and 116. The camera management device112 comprises a communication controller 120 and a storage unit 122,similar to the communication controller 30 and the storage unit 32respectively, and further comprises a camera controller 124 capable ofcontrolling the two or more cameras, and camera interfaces 126 and 128connected to the camera controller 124, each of which is connected thecamera 114 or 116. The same reference numerals in FIG. 13 denote thesame parts as in FIG. 5.

When the single camera management device 112 manages a plurality ofcameras 114 and 116, the camera to be operated can be specified byadding information for specifying one of the cameras 114 and 116 managedby the camera management device 112 in addition to that for specifyingthe camera management device 112.

Of course, when the communication controller 120 of the cameramanagement device 112 can set logic communication channels correspondingin number to the cameras 114 and 116 to be managed, connection requestsfor different cameras from a plurality of camera operation devices 20and 22 can be simultaneously processed.

Similarly, it is easy to extend the functions of the camera operationdevices 20 and 22 to be able to operate a plurality of cameras at thesame time.

In this embodiment, the camera server is built by the camera managementdevice and the camera connected thereto, but may be realized by a singledevice.

As described above, according to the first embodiment, when the viewfield angle of a remote camera is to be changed, the time required forsetting the target angle can be shortened by suppressing thetransmission volume of image data until the target angle is reached inthe camera, and the load on the network can be reduced by suppressingthe image data volume to be transmitted until the target angle isreached in the camera.

[Second Embodiment]

FIG. 14 is a block diagram for explaining the arrangement of an imagecommunication system including an image transmission apparatus accordingto the second embodiment of the present invention.

In FIG. 14, reference numeral 1100 denotes digital transmission mediasuch as a LAN, Internet, ISDN, or the like; and 1200, a camera endsystem (corresponding to the above-mentioned camera management device),which senses an image of the object to be sensed by controlling a cameradevice 1201 and transmits the sensed image data via the digitaltransmission medial 1100. Reference numeral 1300 denotes a viewer endsystem corresponding to the above-mentioned camera operation device(client), which receives image data sent from the camera end system 1200via the digital transmission media 1100, and displays the received dataon a display 1303. The camera device 1201 allows camera control of,e.g., the pan and tilt angles and zoom value.

The arrangement of the camera end system 1200 will be described below.Reference numeral 1202 denotes a compression unit, which compressesimage data sensed by the camera device 1201, and outputs the compresseddigital image data. Reference numeral 1203 denotes a compressionprocessing designation unit for designating the compression mode used bythe compression unit 1202 upon compressing image data. Reference numeral1204 denotes a data transmitter which transmits the image datacompressed by the compression unit 1202 to the viewer end system 1300via the digital transmission media 1100. Reference numeral 1205 denotesa control command receiver which receives a camera control command sentfrom the viewer end system 1300. Reference numeral 1206 denotes a cameracontroller which interprets the control command received by the controlcommand receiver 1205, and controls the operation of the camera device1201 in accordance with the control command.

The arrangement of the viewer end system 1300 will be described below.

Reference numeral 1301 denotes a data receiver, which receives thecompressed image data sent from the camera end system 1200 via thedigital transmission media 1100. Reference numeral 1302 denotes anexpansion unit which expands the compressed image data received by thedata receiver 1301. Reference numeral 1303 denotes a display devicewhich receives and displays the image data expanded by the expansionunit 1302. Reference numeral 1304 denotes a camera control userinterface unit, which interprets user's camera operation, and convertsit into an internal camera control command. Reference numeral 1305denotes a control command transmitter for transmitting the cameracontrol command to the camera end system 1200 via the digitaltransmission medial 1100.

The flow of image data sensed by the camera device 1201 will bedescribed below with reference to FIG. 15.

In step S51, image signals sensed by the camera device 1201 aresequentially supplied to the compression unit 1202 as image data. Inthis case, although the image data may be either an analog signal ordigital signal and may be divisionally sent in units of frames or sentas a bit stream, the image data is preferably sent in a format that canbe easily processed by the compression unit 1202. The compression unit1202 compresses image data by the processing method designated by thecompression processing designation unit 1203. The operation of thecompression processing designation unit 1203 will be described later.

The compression unit 1202 is prepared with a mechanism (function) thatcan select whether or not the compression processing is performed, asshown in step S52, or can selectively execute two different types ofcompression processing, i.e., the processing that assures a highresolution but a low compression ratio of an image, and the processingthat assures a low resolution but a high compression ratio of an image.In order to realize the two types of compression processing, forexample, two different compression schemes (e.g., Motion JPEG andMPEG-1, or the like) may be prepared, or a single compression scheme maybe used by changing the compression parameter (e.g., changing thequantization step in MPEG-1). An identifier that can identify uponexpansion processing whether or not the image data is subjected to thecompression processing (with or without compression), or an identifier(indicating the type of compression) that allows to execute expansionprocessing in correspondence with changes in compression processing isadded to the image data compressed by the compression unit 1202. Notethat the compression unit 1202 may be implemented by either hardware orsoftware.

The image data compressed by the compression unit 1202 is transmitted bythe data transmitter 1204 to the viewer end system 1300 via the digitaltransmission media 1100 (S53). A protocol suitable for thespecifications of the digital transmission media used is preferably usedupon transmission of the image data. In general, when image data istransmitted, a data gram method with a small transmission overhead isused. For example, in the case of a computer network such as a LAN,Internet, or the like, UDP with a smaller overhead than TCP is normallyused.

The compressed image data transmitted via the digital transmission media1100 is received by the data receiver 1301 (S54). The received imagedata is further supplied to the expansion unit 1302. If it is determinedbased on the identifier added to the image data if the data iscompressed (S55), the compressed image data is expanded by an expansionmethod corresponding to the compression method (S56). The expansion unit1302 does not execute any expansion processing is if it detects based onthe identifier that the data is not compressed. Also, assume that theexpansion unit 1302 has an expansion processing function correspondingto all the kinds of expansion processing of the compression unit 1202 inthe camera end system 1200. The image data which is expanded by theexpansion unit 1302 or the uncompressed image data is output to anddisplayed on the display device 1303 (S57).

The flow of processing executed when the user makes camera control usingthe viewer end system 1300 will be explained below with reference toFIG. 16.

The user starts a camera control operation via the camera control userinterface unit 1304 (S61). The camera control user interface unit 1304may allow the pan, tilt, and zoom operations by operating mechanicalswitches or may be implemented by a graphical user interface on acomputer and by operating control bars displayed on, e.g., the displaydevice 1303 using, e.g., a mouse cursor. The user can control theoperation of the camera device 1201 via the camera control userinterface unit 1304 while observing an image that changes incorrespondence with the camera operation.

The camera control user interface unit 1304 converts a camera operationinput by the user's operation into a control command (S62), and sendsthat control command to the control command transmitter 1305. Thecontrol command may include commands “start clockwise rotation”, “startcounterclockwise rotation”, and “stop rotation”, or commands “rotate 30°clockwise”, “rotate 30° counterclockwise”, and the like. In order toidentify such control commands, unique numerical values are assigned tothe individual commands. For example, the command “rotate 30° clockwise”may be expressed by a combination of a command “rotate clockwise” and aparameter “30°”. In the second embodiment, the command expression methodis not specifically limited.

The control command transmitter 1305 transmits the control commandreceived from the camera control user interface unit 1304 to the cameraend system 1200 via the digital transmission media 1100 (S63). Since areliable communication protocol is suitable for transmitting the controlcommand unlike image data, for example, a protocol call TCP ispreferably used in the case of a computer network such as a LAN,Internet, or the like.

The transmitted control command is received by the control commandreceiver 1205 of the camera end system 1200 (S64). The control commandreceiver 1205 sends the received control command to the cameracontroller 1206. The camera controller 1206 interprets the controlcommand sent from the control command receiver 1205 (S65), and controlsthe camera device 1201 in accordance with the instruction of the controlcommand (S66).

The operation of the camera controller 1206 will be described in detailbelow with reference to the flow chart in FIG. 17.

When the camera controller 1206 interprets the control command sent fromthe control command receiver 1205 in step S71, and starts camera controlin accordance with the command (S72), it informs the compressionprocessing designation unit 1203 that the camera control is in progress.Upon completion of the camera control, the controller 1206 informs thecompression processing designation unit 1203 to that effect. With thisinformation, the compression processing designation unit 1203 detectswhether or not the camera control is in progress, and designates thecompression processing method to the compression unit 1202 in accordancewith the detection result.

That is, in a normal state in which no camera control is done, the flowadvances from step S72 to step S74, and the compression processingdesignation unit 1203 instructs the compression unit 1202 to docompression processing by a compression method initially set by thecamera end system 1200 or the user (or to revert to the initialcompression method). On the other hand, if it is detected that thecamera control is in progress, the flow advances form step S72 to stepS73, and if the compression processing has not been performed so far,the compression processing designation unit 1203 instructs thecompression unit 1202 to start compression processing. On the otherhand, if the compression unit 1202 has the above-mentioned two kinds ofcompression processing, and has executed compression processing thatassures a high resolution but a low compression ratio of an image, theunit 1203 instructs the compression unit 1202 to switch the compressionprocessing to the one that assures a low resolution but a highcompression ratio of an image. If it is detected that the camera controlis complete, the unit 1203 instructs the compression unit 1202 torestore an initial state in step S74.

As described above, according to the second embodiment, even when theuser normally observes an image with a low frame rate and a highresolution, since the compression processing is automatically switchedto a compression scheme that can increase the frame rate (a compressionscheme with a high compression ratio) during the camera control, theuser can recognize the camera control state in real time by watching thedisplayed image.

Also, according to the second embodiment, the compression processing isautomatically switched to a compression scheme that can set a highcompression ratio and frame rate during the camera control, and isreverted to an initial state, i.e., to an initial compression method, orthe compression processing is stopped upon completion of the cameracontrol. Hence, the user can easily recognize the control state duringthe camera control while he or she normally observes an image with ahigh resolution.

[Third Embodiment]

FIG. 18 is a block diagram for explaining the arrangement of an imagecommunication system according to the third embodiment of the presentinvention. Reference numerals 310 and 320 denote communicationterminals, terminal main bodies 311 and 321 of which comprise CPUs 341and 351, ROMs 342 and 352, RAMs 343 and 353, external storage devices344 and 354, and the like as in a normal computer. The communicationterminals 310 and 320 are connected via a network 330 to communicatewith each other. Note that the communication terminal 310 serves as acamera management (control) device, and the communication terminal 320serves as a client.

Reference numeral 312 denotes a video camera. Reference numeral 313denotes an image input unit which captures an analog image signal outputfrom the video camera 312 and converts it into digital data. Referencenumeral 314 denotes a camera controller, which controls the pan and tiltangles, zoom value, and the like of the video camera 312 (such controlwill be referred to as position control hereinafter). Reference numeral315 denotes a network transmitter/receiver, which transmits a packetonto the network 330 and receives a packet flowing on the network 330.The arrangements of the above-mentioned image input unit 313, cameracontroller 314, and network transmitter/receiver 315 are realized byhardware components complying with their arrangements and byappropriately controlling these hardware components by executing acontrol program stored in the ROM 342 or RAM 343 by the CPU 341.

The arrangement of the communication terminal 320 on the client sidewill be explained below.

Reference numeral 327 denotes an image buffer which stores digital imagedata received via the network 330. The image buffer 327 uses the RAM 353or the external storage device 354 to make up a First-In, First-Outmemory that can read out initially stored data first. Reference numeral322 denotes a display (monitor) which performs various kinds of displayunder the control of the CPU 351. The received image is also displayedon the display 322. Reference numeral 328 denotes a mouse as a pointingdevice; and 329, a keyboard.

Reference numeral 323 denotes a camera control request unit, whichconverts an instruction of the pan or tilt angle, zoom value, or thelike input by the mouse 328 or keyboard 329 into a command, andtransfers the command to a network transmitter/receiver 324 to transmitit to the communication terminal 310. Reference numeral 326 denotes abuffer suppression unit, which instructs the image buffer 327 to bufferimage data or to directly display the image data on the display 322without storing the image data into the image buffer 327. Referencenumeral 325 denotes a camera control detector, which detects if a cameracontrol request is issued, and supplies the detection result to thebuffer suppression unit 326. The network transmitter/receiver 324transmits a packet onto the network 330, and receives a packet flowingon the network 330.

The above-mentioned units denoted by reference numerals 323 to 327 arerealized by hardware components complying with their arrangements and byappropriately controlling these hardware components by executing acontrol program stored in the ROM 352 or RAM 353 by the CPU 351.

FIG. 19 shows a display example on the display 322 at the communicationterminal 320 (client) of the third embodiment.

In FIG. 19, reference numeral 3301 denotes a mouse cursor displayed onthe display 322. As is well known, the mouse cursor 3301 can freely moveon the screen upon operation of the mouse 328. Reference numeral 3302denotes an image displayed on the display 322, which is sent from thevideo camera 312 of the communication terminal 310. Reference numeral3303 denotes a user interface for a camera operation.

The operation until the communication terminal 310 outputs image datawill be explained below with reference to FIG. 18.

Analog image data output from the video camera 315 of the communicationterminal 310 is converted into digital data by the image input unit 313.The digital image data is packetized by the network transmitter/receiver315, and the obtained packet is output from the communication terminal310 onto the network 330 at predetermined periods.

The operation from when the communication terminal 320 receives imagedata until it displays that image data will be described below withreference to FIG. 18 and FIGS. 20 and 21. FIG. 20 is a block diagramshowing the image buffer 327 of the third embodiment and its peripheralportions in detail. FIG. 21 is a flow chart showing the display sequenceof image data received by the communication terminal 320 of the thirdembodiment.

As shown in FIG. 20, the image buffer 327 comprises a buffer memory 327a, a buffer controller 327 b, and an output controller 327 c. The buffermemory 327 a is assured by using a portion of, e.g., the RAM 353, andcan store image data for a maximum of 10 frames in the third embodiment.The buffer controller 327 b buffers input image data in the buffermemory 327 a when a suppression signal 331 from the buffer suppressionunit 326 is OFF, or sends the input image data to the output controller327 c when the suppression signal 331 is ON. When the suppression signal331 is OFF, the output controller 327 c reads out the buffered data fromthe buffer memory 327 a, and outputs it to a drawing unit 355; when thesuppression signal 331 is ON, it outputs image data from the buffercontroller 327 b to the drawing unit 355. As a result, when thesuppression signal 331 is ON, input image data is directly output to thedrawing unit 355 without being stored in the buffer memory 327 a.

The drawing unit 355 draws an image on a VRAM 356 on the basis of theinput image data. The image drawn on the VRAM 356 is displayed on thedisplay 322 under the control of a video controller (not shown).

The operation of the buffer suppression unit 326 and the image buffer327 will be explained in more detail below with reference to the flowchart in FIG. 21.

It is checked in step S81 if a packet of image data is received. If YESin step S81, the flow advances to step S82 to check if the cameracontrol is in progress. This checking step is attained by checking ifthe suppression signal 331 from the buffer suppression unit 326 is ON,as described above. If NO in step S82, the flow advances to step S83,and the received image data is buffered in the buffer memory 327 a.

It is checked in step S84 if image data for 10 frames are stored in thebuffer memory 327 a, or if a display flag is ON. If image data for 10frames are buffered, the display flag is turned on to start a display ata predetermined display timing (step S85). If the display timing hasbeen reached in step S86, image data for the oldest frame is read outfrom the buffer memory 327 a, and is drawn on the display 322 (stepS87). Note that the readout image data is deleted from the buffer memory327 a. Thereafter, while the display flag is ON, buffering of receivedimage data in the buffer memory 327 a and display of image data read outfrom the buffer memory 327 a are parallelly performed. After all theimage data are read out from the buffer memory 327 a, the display flagis turned off (steps S88 and S89).

When the camera control request unit 323 issues a camera control requestduring the above-mentioned processing, the camera control detector 325detects that request, and informs the buffer suppression unit 326 of it.When the buffer suppression unit 326 turns on the suppression signal331, the buffer controller 327 b transfers input image data to theoutput controller 327 c without going through the buffer memory 327 a.When the output controller 327 c sends the image data to the drawingunit 355, the received image data is displayed without being buffered(steps S82 and S90).

With the above arrangement, image data can be inhibited from beingbuffered during a camera operation, and an image that has changed by thecamera operation can be displayed without any delay time. For thisreason, an image can be displayed in real time, and the camera operationis facilitated.

Note that the above description has exemplified a communication systemin which the communication terminals have a one-to-one connection.However, the present invention is not limited to such specific system,but can also be applied to a communication system in which thecommunication terminals have a many-to-many connection. Thecommunication terminals of the third embodiment are classified into atransmission dedicated terminal (communication terminal 310) and adisplay dedicated terminal (communication terminal 320). However, thepresent invention is not limited to this, but may be applied tocommunication terminals which communicate with each other.

As described above, according to the third embodiment, when the camerais remote-controlled by the client device via the network, an imagesensed upon executing the position control of the camera is displayed inreal time without being buffered, and the displayed image responds wellto the position control of the camera, thus improving the operability ofthe camera.

[Fourth Embodiment]

In the third embodiment, when the image buffer 327 has already storedimage data upon operation of the camera, the latest image cannot bedisplayed until all the buffered image data are read out after cameraoperation. The fourth embodiment can solve such problem.

FIG. 22 is a block diagram showing the arrangement of an imagecommunication system according to the fourth embodiment of the presentinvention. The same reference numerals in FIG. 22 denote the same partsas those in the arrangement shown in FIG. 18, and a detailed descriptionthereof will be omitted.

Reference numeral 361 denotes a buffer clearing unit, which clears imagedata buffered in the image buffer 327 when the camera control detector325 detects user's camera operation.

FIG. 23 is a block diagram showing the arrangement of the image buffer327 in the fourth embodiment and its peripheral portions in detail. Thesame reference numerals in FIG. 23 denote the same parts as in the thirdembodiment (FIG. 20). In the fourth embodiment, the buffer clearing unit361 is arranged to simultaneously clear the image data stored in thebuffer memory 327 a.

In the above-mentioned arrangement, the console window shown in FIG. 19is displayed on the display 322 of the communication terminal 320, andwhen the mouse cursor 3301 is moved on the screen using the mouse 328,the position control of the video camera 312 can be done using thecamera control interface 3303. At this time, the control information isinput to the camera control request unit 323. The camera control requestunit 323 converts that request into a command, and transfers the commandto the network transmitter/receiver 324. The networktransmitter/receiver 324 converts the command into a packet, andtransmits the packet onto the network 330. Upon reception of the packet,the network transmitter/receiver 315 of the communication terminal 310converts the packet into the command again, and transfers the command tothe camera controller 314. The camera controller 314 controls the camera312 in accordance with the command.

FIG. 24 is a flow chart for explaining the processing sequence of thebuffer clearing unit 361 of the fourth embodiment.

When a camera control request is supplied to the camera control requestunit 323 of the communication terminal 320, the camera control detector325 detects that request. Upon detection of the camera control requestinput from the camera control request unit 323, the camera controldetector 325 informs the buffer clearing unit 361 of a messageindicating it. Upon reception of the message, the buffer clearing unit361 checks if image data are stored in the buffer memory 327 a of theimage buffer 327 (steps S91 and S92). If image data are stored in thebuffer memory 327 a, the buffer clearing unit 361 clears all the imagedata stored in the buffer memory 327 b (step S93).

With the above-mentioned arrangement, image data buffered upon cameraposition control is not displayed after the position control iscomplete, and an image can be displayed immediately after the cameraposition control is complete. In this manner, easy camera operationcontrol is attained.

Note that the fourth embodiment has exemplified a communication systemin which the communication terminals have a one-to-one connection, as inthe third embodiment. However, the present invention can also be appliedto a communication system in which the communication terminals have amany-to-many connection. The communication terminals of the thirdembodiment are classified into a transmission dedicated terminal(communication terminal 310) and a display dedicated terminal(communication terminal 320). However, the present invention is notlimited to this, but may be applied to communication terminals whichcommunicate with each other.

As described above, according to the fourth embodiment, the client canquickly display image data sensed by the camera at the updated positionupon completion of the camera control by the client, in addition to theeffects of the third embodiment described above.

In the third and fourth embodiments, when the communication terminal onthe client side issues a position control instruction of the camera, theclient detects the control instruction and suppresses the buffering.However, when the video camera 312 receives the position controlinstruction, the transmitter terminal of image data may add informationindicating that message to image data and transmit the image data, andthe client may detect based on the information added to the image datathat the camera position control is in progress. In this case, forexample, the camera control detector 325 separates the additionalinformation from the received image data packet, and detects if thecamera position control is in progress. With this control, when aplurality of clients are connected, even a client that does not executethe camera position control can display an image that follows theposition of the camera.

When the program of this embodiment is applied to a storage medium, thestorage medium stores program codes corresponding to the flow chartsdescribed above. Briefly speaking, the storage medium stores modulesshown in a memory map example of FIG. 25. That is, the storage mediumstores program codes of, e.g., a “reception processing module”, “bufferprocessing module”, “first display processing module”, “detectionprocessing module”, and “second display processing module”. Preferably,the storage medium stores a program code of a “clearing processingmodule” in addition to the above-mentioned program codes.

These processing modules make up a control program for receiving imageinformation from an external device on the network, and displaying thereceived image information, and the reception processing module executesreception processing for receiving image information. The bufferprocessing module implements buffer processing for buffering thereceived image information. The first display processing moduleimplements first display processing for displaying an image on the basisof the image information stored by the buffer processing. The detectionprocessing module implements detection processing for detecting if animage sensing device that is capturing the image information transmittedfrom the external device is being controlled. The second displayprocessing module implements second display processing for directlydisplaying image information received by the reception processingwithout going through the buffer processing when the detectionprocessing detects that the image sensing device is being controlled.Furthermore, the clearing processing module implements clearingprocessing for clearing image information stored by the bufferprocessing when the detection processing detects that the image sensingdevice is being controlled.

[Fifth Embodiment]

In the fifth embodiment, if a client which is receiving and displaying acamera image has the right of access to that camera, it can change theprocessing method of an image signal (including media data that includesimage and audio data, and the like) to be transmitted via the network.That is, if the client which is receiving an image from the camera hasthe right of access to that camera, the image is subjected tolow-compression coding with less image deterioration, and is transmittedto that client; if the client does not have any right of access,high-compression-coded image data is transmitted to that client.

FIG. 26 is a block diagram showing the arrangement of an imagecommunication system according to the fifth embodiment of the presentinvention. In this embodiment, one or a plurality of camera serverdevices 5100 and a plurality of client devices 5200 are connected via anetwork 5210.

In FIG. 26, reference numeral 5100 denotes a camera server device(camera management device); and 5200, a client device (a display controlterminal that can be operated by the user). The camera server device5100 captures an image signal from a video camera 5103, and distributesthe image signal to clients via the network 5210. Upon reception of acamera control command from each client via the network 5210, the cameraserver device 5100 controls the position and operation of the videocamera 5103 in accordance with the received command. In the fifthembodiment, the number of client devices connected to the network 5210is not particularly limited.

The client device 5200 requests the camera server device 5100 todistribute an image, and observes an image from the camera server device5100. Also, the client device 5200 issues a camera control request tothe camera server device 5100 to execute camera control. In thisembodiment, the individual clients can control the video camera 5103 ofthe camera server device 5100 in turn, but are exclusively controlled,so that only one client device connected to the network 5210 is grantedthe right of access and can control the camera 5103.

The arrangement of the camera server device 5100 will be explainedbelow.

The camera server device 5100 comprises the video camera 5103, a cameracontroller 5105 for controlling the pan and tilt angles of the panpod ofthe video camera 5103, the zoom ratio of the camera 5103, and the like,a video capture unit 5104 for capturing an image from the video camera5103, a network interface (I/F) 5107 for distributing image datacaptured by the video capture unit 5104 onto the network 5210, a CPU5101 which controls the individual units and includes a main memory unit5102, and a secondary storage unit 5106 for storing a control programand various data. Note that the camera controller 5105 receives a cameracontrol command sent from the client device 5200 via the network I/F5107, and can control the pan and tilt angles, zoom ratio, and the likeof the video camera 5103. The video capture unit 5104 captures an imagesignal from the video camera 5103, A/D-converts the captured imagesignal, compresses the digital video data, and then transfers it to thenetwork I/F 5107. As the image compression scheme, Motion JPEG or thelike may be used, but the present invention is not limited to a specificscheme. Reference numeral 5108 denotes a system bus for connecting theabove-mentioned units.

The arrangement of the client device 5200 will be described below.

The compressed image data distributed from the camera server device 5100is received via a network interface (I/F) 5204, is expanded under thecontrol of a CPU 5201, and is displayed on the screen of a display 5203.Note that a user interface window for issuing an acquisition request ofthe right of access is displayed on the screen of the display 5203 inaddition to a window for displaying the received image. Referencenumeral 5202 denotes a main memory unit for storing a control programexecuted by the CPU 5201, and the like; 5206, an input unit comprising apointing device such as a mouse, a keyboard, and the like; and 5205, asecondary storage unit such as a hard disk or the like, which storesvarious programs, data, and the like executed by the CPU 5201. Referencenumeral 5207 denotes a system bus for connecting these units.

FIG. 27 is a block diagram showing the arrangement of the video captureunit 5104 of the fifth embodiment.

The video capture unit 5104 comprises a selector 5301 for selecting oneof compression units, which compresses an image signal from the videocamera 5103, a compression unit 5302 for performing high-compressioncoding, and a compression unit 5303 for performing low-compressioncoding.

More specifically, when an image signal is transmitted to a clientdevice which has no right of access to the video camera 5103, theselector 5301 supplies the image signal to the compression unit 5302,which performs high-compression coding of the input image signal. Withthis control, an image which has a small coded data volume but low imagequality upon reproduction is transmitted to the client device which hasno right of access. In contrast to this, for a client device having theright of camera access, the selector 5301 selects the compression unit5303, which performs low-compression coding of the image signal, and thecoded signal is transmitted. With this control, the client device whichhas the right of access can receive the compressed image signal, whichhas high image quality upon reproduction although it has a large codedata volume.

In this manner, a normal image is reproduced on the client device havingno right of camera access, and a high-quality image that allows easycamera control is reproduced and displayed on the client device whichhas the right of camera access and makes camera control.

FIG. 29 is a view for explaining the format of a transmissiondestination table 5111 assured on the main memory unit 5102 of thecamera server device 5100 of the fifth embodiment.

The transmission destination table 5111 registers client names to whichimages are transmitted, and the “presence/absence” of the right ofcontrol in correspondence with the client names.

FIG. 30 shows the data format of image data to be transmitted from thecamera server device 5100 to the client device 5200. A transmissiondestination address 5500 is stored, and corresponding image data 5501 isstored.

FIG. 31 is a flow chart showing the processing in the camera serverdevice 5100 of the fifth embodiment. A control program that executesthis processing is stored in the main memory unit 5102. Note that thisprogram may be stored in the secondary storage unit 5106, and may beloaded onto the main memory unit 5102 when it is executed.

This processing is started upon transmission of an image to the clientdevice in response to an image signal request from the client device5200, generation of a timer event, or the like. In step S101, it ischecked if the client device 5200 of interest has the right of access.If YES in step S101, the flow advances to step S102, and the selector5301 in FIG. 27 outputs the image signal from the video camera 5103 tothe compression unit 5303 to perform low-compression coding of the imagedata. The compressed image signal is transmitted to the client device5200 in the format shown in FIG. 30 via the network I/F 5107 and thenetwork 5210.

On the other hand, if the client device 5200 of interest has no right ofaccess, the flow advances to step S104, and the selector 5301 selectsthe compression unit 5302 to compress the image signal. The compressedimage signal is transmitted to the client device in the format shown inFIG. 30 via the network I/F 5107 and the network 5210 (S103).

[Sixth Embodiment]

FIG. 28 is a block diagram showing the functional arrangement of aclient device 5200 a according to the sixth embodiment of the presentinvention.

The client device 5200 a comprises a selector 5310 which selects one ofjitter absorption buffers used upon reception of image data transmittedvia the network I/F 5107 and the network 5210, a large-capacity jitterabsorption buffer 5311, and a small-capacity jitter absorption buffer5312.

The camera server device 5100 transmits a control packet that instructsto use the large-capacity jitter absorption buffer to the client devicehaving no right of access to the video camera 5103, and also transmitsimage data. In response to this control packet, the client device 5200 aselects the relatively large-capacity jitter absorption buffer 5311using the selector 5310. In this manner, on the client device 5200 ahaving no right of access, an image with a long delay time is decodedand displayed. On the contrary, the camera server device 5100 transmitsa control packet that instructs to use the small-capacity jitterabsorption buffer 5312 to the client device 5200 a having the right ofcamera access, and transmits corresponding image data. In response tothis control packet, the client device 5200 a selects the small-capacityjitter absorption buffer 5312 using the selector 5310. In this manner,the client device 5200 a with the right of access can reduce the buffercapacity for jitter absorption, and can reproduce and display image datadecoded by an image decoder 5313 (the video capture unit 5104). Withthis control, on the client device 5200 a that makes camera control, animage with a short response time, i.e., a short delay time can bereproduced and displayed.

FIG. 32 is a flow chart showing the processing in the client device 5200a according to the sixth embodiment of the present invention. Thisprocessing program is stored in the main memory unit 5202. Note thatthis program may also be stored in the secondary storage unit 5205 andmay be loaded onto the main memory unit 5202 when it is executed. Thisprocessing is started when the camera server device 5100 starts inresponse to an image signal request from the client device.

Upon reception of a control packet addressed to the own terminal, it ischecked if the own terminal has the right of camera access (S111). IfYES in step S111, an image signal received from the camera server device5100 is stored in the small-capacity jitter absorption buffer 5312(S112). The image data stored in the buffer 5312 is decoded in units offrames (S113), and the decoded image data is displayed on the display5203 (S115). This control is to end upon completion of communications orupon expiration of a predetermined camera access right period. On theother hand, if it is determined based on the control packet addressed tothe own terminal that the own terminal has no right of camera access(S111), the large-capacity jitter absorption buffer 5311 is used (S114).

As described above, according to the fifth embodiment, since theprocessing method of image data addressed to the client terminal withthe right of camera access is controlled to be different from that ofimage data addressed to the client terminal with no right of cameraaccess, the camera control can be facilitated.

Also, according to this embodiment, the client device having no right ofcamera access can display and confirm an image at high speed, and theclient device having the right of camera access can display an imagethat allows easy camera control.

Note that the present invention may be applied to either a systemconstituted by a plurality of equipments (e.g., a host computer, aninterface device, a reader, a printer, and the like), or an apparatusconsisting of a single equipment (e.g., a copying machine, a facsimileapparatus, or the like).

The objects of the present invention are also achieved by supplying astorage medium, which records a program code of a software program thatcan realize the functions of the above-mentioned embodiments to thesystem or apparatus, and reading out and executing the program codestored in the storage medium by a computer (or a CPU or MPU) of thesystem or apparatus.

In this case, the program code itself read out from the storage mediumrealizes the functions of the above-mentioned embodiments, and thestorage medium which stores the program code constitutes the presentinvention.

As the storage medium for supplying the program code, for example, afloppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM,CD-R, magnetic tape, nonvolatile memory card, ROM, and the like may beused.

The functions of the above-mentioned embodiments may be realized notonly by executing the readout program code by the computer but also bysome or all of actual processing operations executed by an OS (operatingsystem) running on the computer on the basis of an instruction of theprogram code.

Furthermore, the functions of the above-mentioned embodiments may berealized by some or all of actual processing operations executed by aCPU or the like arranged in a function extension board or a functionextension unit, which is inserted in or connected to the computer, afterthe program code read out from the storage medium is written in a memoryof the extension board or unit.

In the above description, the individual embodiments have beenindependently explained for the sake of simplicity. However, the presentinvention is not limited to such specific embodiments, and may beapplied to appropriate combinations of he arrangements of theabove-mentioned embodiments.

The present invention is not limited to the above embodiments andvarious changes and modifications can be made within the spirit andscope of the present invention. Therefore, to apprise the public of thescope of the present invention, the following claims are made.

What is claimed is:
 1. An image transmission system including an imagesensing control apparatus for transmitting an image signal sensed by animage sensing device to a client via a network, and a plurality ofclients, the image sensing control apparatus comprising: determinationmeans for determining whether or not a client being a destination of theimage signal has a right of an operation of the image sensing device;process control means for selecting image quality of the image signalfrom the image sensing device in accordance with a determination resultof said determination means and processing the image data based on theimage quality; and transmission means for transmitting the image signalprocessed by said process control means to the client via the network,wherein the client requests the image signal sensed by the image sensingdevice and acquires of the right of an operation of the image sensingdevice.
 2. The system according to claim 1, wherein said process controlmeans selects the image quality of high-compression coding of the imagesignal when said determination means determines that the client does nothave the right of access.
 3. The system according to claim 1, whereinsaid process control means selects the image quality of low-compressioncoding of the image signal when said determination means determines thatthe client does not have the right of access.
 4. An image transmissionsystem including an image sensing control apparatus for transmitting animage signal sensed by an image sensing device to a client via anetwork, and a plurality of clients, the image sensing control apparatuscomprising: transmission means for transmitting an image signal to aclient; and storage capacity control means for instructing a storagecapacity of a buffer for temporarily storing the image signal in theclient in accordance with whether the client has the right to access theimage sensing device, wherein the client switches the storage capacityof the buffer for temporarily storing the image signal in accordancewith an instruction instructed by said storage capacity control means.5. The system according to claim 4, wherein said storage capacitycontrol means instructs the client to use a large-capacity jitterabsorption buffer when the client has no right to access the imagesensing device.
 6. The apparatus according to claim 4, wherein saidstorage capacity control means instructs the client to use asmall-capacity jitter absorption buffer when the client has the right toaccess the image sensing device.
 7. An image transmission method in animage transmission system including an image sensing control apparatusfor transmitting an image signal sensed by an image sensing device to aclient via a network, and a plurality of clients, the method comprising:a determination step of determining whether or not a client being adestination of an image signal has a right of an operation of the imagesensing device; a process control step of selecting image quality of theimage signal from the image sensing device in accordance with adetermination result in said determination step and processing the imagedata based on the image quality; and a transmission step of transmittingthe image signal processed in said process control step to the clientvia the network, wherein the client requests the image signal sensed bythe image sensing device and acquires of the right of an operation ofthe image sensing device.
 8. The method according to claim 7, wherein insaid process control step, the image quality of high-compression codingof the image signal is selected when it is determined that the clientdoes not have the right of access in said determination step.
 9. Themethod according to claim 7, wherein in said process control step, theimage quality of low-compression coding of the image signal is selectedwhen it is determined that the client has the right of access in saiddetermination step.
 10. An image transmission method in an imagetransmission system including an image sensing control apparatus fortransmitting an image signal sensed by an image sensing device to aclient via a network, and a plurality of clients, the method comprising:a transmission step of transmitting an image signal to a client; aninstruction step of instructing a storage capacity of a buffer fortemporarily storing the image signal in the client in accordance withwhether the client has the right to access the image sensing device, anda step of switching the storage capacity of the buffer for temporarilystoring the image signal in accordance with an instruction instructed insaid instruction step.
 11. The method according to claim 10, wherein insaid instruction step, the client is instructed to use a large-capacityjitter absorption buffer when the client has no right to access theimage sensing device.
 12. The method according to claim 10, wherein insaid instruction step, the client is to use a small-capacity jitterabsorption buffer when the client has the right to access the imagesensing device.
 13. An image sensing control apparatus for transmittingan image signal sensed by an image sensing device to a client via anetwork, comprising: determination means for determining whether or nota client being a destination of the image signal has a right of anoperation of the image sensing device, in a case where the clientrequests an image signal sensed by the image sensing device; processcontrol means for selecting image quality of the image signal from theimage sensing device in accordance with a determination result of saiddetermination means and processing the image data based on the imagequality; and transmission means for transmitting the image signalprocessed by said process control means to the client via the network.14. The apparatus according to claim 13, wherein said process controlmeans selects the image quality of high-compression coding of the imagesignal when said determination means determines that the client does nothave the right of access.
 15. The apparatus according to claim 13,wherein said process control means selects the image quality oflow-compression coding of the image signal when said determination meansdetermines that the client has the right of access.
 16. An image sensingcontrol apparatus for transmitting an image signal sensed by an imagesensing device to a client via a network, comprising: transmission meansfor transmitting an image signal to a client in response to a requestfrom the client; and storage capacity control means for instructing astorage capacity of a buffer for temporarily storing the image signal inthe client in accordance with whether the client has the right to accessthe image sensing device.
 17. An image transmitting apparatus fortransmitting an image signal sensed by an image sensing device to aclient via a network, comprising: determination means for determiningwhether or not a client has a right of an operation of the image sensingdevice; generation means for generating an image signal having an imagequality in accordance with a determination result of said determinationmeans; and transmission means for transmitting the image signalgenerated by said generation means to the client.
 18. The apparatusaccording to claim 17, wherein said generation means generates an imagesignal having the image quality of high-compression coding when saiddetermination means determines that the client does not have the rightof access.
 19. The apparatus according to claim 17, wherein saidgeneration means generates an image signal having the image quality oflow-compression coding when said determination means determines that theclient has the right of access.
 20. A method of transmitting an imagesignal sensed by an image sensing device to a client via a network,comprising the steps of: determining whether or not a client has a rightof an operation of the image sensing device; generating an image signalhaving an image quality decided in accordance with a determinationresult in said determining step; and transmitting the image signalgenerated in said generating step to the client.
 21. The methodaccording to claim 20, wherein in said generation step, the image signalhaving the image quality of high-compression coding is generated in saidgenerating step when it is determined that the client does not have theright of access in said determining step.
 22. The method according toclaim 20, wherein in said generation step, the image signal having theimage quality of low-compression coding is generated in said generatingstep when it is determined that the client has the right of access insaid determining step.